Live 3-Dimensional Content for Augmented Reality

We describe an augmented reality system for superimposing three-dimensional live content onto two-dimensional fiducial markers in the scene. In each frame, the Euclidean transformation between the marker and the camera is estimated. The equivalent “virtual view” of the live model is then generated and rendered into the scene at interactive speeds. The three-dimensional structure of the model is calculated using a fast shape-from-silhouette algorithm based on the outputs of 15 cameras surrounding the subject. The novel view is generated by projecting rays through each pixel of the desired image and intersecting them with the 3-d structure. Pixel color is estimated by taking a weighted sum of the colors of the projections of this three-dimensional point in nearby real camera images. Using this system, we capture live human models and present them via the augmented reality interface at a remote location. We can generate 384x288 pixel images of the models at 25 fps, with a latency of <100ms. The result gives the strong impression that the model is a real three-dimensional part of the scene.